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Abstract 

This  paper  reports  the  results  of  research  on  whether  changes  in  Department  of  Defense 
(DoD)  acquisition  policy  and  process  have  had  a  discernible  effect  on  the  growth  of  Program 
Acquisition  Unit  Cost  (PAUC)  of  major  defense  acquisition  programs  (MDAPs).  Examination 
of  PAUC  growth  data  for  151  MDAPs  that  passed  Milestone  II  or  B  during  fiscal  year  (FY) 
1970-FY  2007  does  not  reveal  any  substantial  or  consistent  effect  of  changes  in  acquisition 
policy  and  process.  Changes  in  funding  climate,  however,  are  found  to  have  a  large  influence 
on  PAUC  growth.  These  findings  have  three  implications  for  acquisition  reform.  First,  the 
relevant  context  for  understanding  PAUC  growth  is  the  interface  between  the  acquisition 
process  and  the  program/funding  process.  Second,  it  seems  unlikely  that  further  changes  in 
the  acquisition  process  will  have  a  major  effect  on  PAUC  growth.  Third,  the  underlying  cause 
of  persistent  high  PAUC  growth  is  not,  as  is  commonly  asserted,  a  deeply  established  culture 
of  the  DoD  acquisition  organizations  and  their  professional  employees. 

Introduction 

This  paper  reports  the  results  of  research  on  whether  changes  in  Department  of 
Defense  (DoD)  acquisition  policy  and  process  have  had  a  discernible  effect  on  the  growth  of 
Program  Acquisition  Unit  Cost  (PAUC)  of  major  defense  acquisition  programs  (MDAPs).1  A 
few  previous  studies  have  broached  this  topic,  but  it  has  received  little  attention,  and  the 
work  that  has  been  done  has  not  resulted  in  any  accepted  or  even  widely  recognized 
conclusions.2 


1  The  text  of  this  paper  is  substantially  identical  to  that  of  David  L.  McNicol  and  Linda  Wu  (2014), 
Evidence  on  the  Effect  of  DoD  Acquisition  Policy  and  Process  on  Cost  Growth  of  Major  Defense 
Acquisition  Programs.  The  appendices  of  P-5126  and  its  associated  data  CD  are  not  included  here, 
however.  A  PDF  version  of  the  paper  is  available  at 

https://www.ida.Org/~/media/Corporate/Files/Publications/IDA  Documents/CARD/P-5126.ashx.  To 

obtain  a  hard  copy,  contact  IDA  Library  Reference  Services  at  (703)  845-2087  or  refdesk@ida.org.  A 
summary  of  P-5126  is  included  in  Defense  Acquisition  Reform:  Where  Do  We  Go  From  Here?  (2014). 

2  The  literature  includes  many  analyses  of  particular  acquisition  policies.  There  also  have  been 
several  largely  qualitative  studies  of,  for  example,  the  extent  to  which  a  set  of  related  initiatives  has 
been  successful  or,  to  offer  another  example,  the  apparent  successes  and  shortcomings  of  the 
acquisition  process  over  a  specific  time  period,  such  as  a  decade.  In  contrast,  there  have  been  few 
broad  quantitative  assessments  of  the  effectiveness  of  acquisition  policy  and  process.  The  main 
predecessors  of  this  work,  in  particular,  are  David  L.  McNicol  (2005),  Cost  Growth  in  Major  Weapon 
Procurement  Programs,  2nd  ed.,  especially  pages  41-44  and  55-59,  and  Tyson  et  al.  (1992),  in  The 
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The  establishment  of  the  Defense  Systems  Acquisition  Review  Council  (DSARC)  in 
late  fiscal  year  (FY)  1 969  marks  the  start  of  systematic  Office  of  the  Secretary  of  Defense 
(OSD)  oversight  of  MDAPs.  While  an  initiative  of  then  Deputy  Secretary  of  Defense  David 
Packard,  the  DSARC  responded  to  intense  congressional  concerns  with  growth  in  the  costs 
of  major  DoD  weapon  system  acquisition  programs.  It  appears  to  have  generally  been 
regarded  at  the  time  as  a  successful  innovation. 

There  have  been  many  changes,  large  and  small,  in  DoD  acquisition  policy  and 
process  since  the  DSARC  was  established.  Many  of  these  were  undertaken  simply  for 
reasons  of  good  government — to  reduce  the  costs  of  the  decision-making  process  and  the 
time  it  requires,  to  increase  its  transparency,  to  make  it  more  responsive  to  policy  direction, 
and  to  adapt  it  to  changes  in  the  technological  and  national  security  environment.  Many 
others  were  aimed  directly  at  improving  outcomes  on  MDAPs — in  particular,  reducing  cost 
growth.  This  study  was  undertaken  in  the  hope  that  a  better  understanding  of  the  effects  of 
these  changes  in  acquisition  policy  and  process  on  PAUC  growth  will  contribute  to  the  long- 
running  discussion  of  reform  of  the  DoD  acquisition  process. 

Finding  or  making  estimates  of  PAUC  growth  for  a  sufficiently  large  set  of  MDAPs 
was  the  first  major  challenge  faced  by  this  study.  We  have  a  PAUC  growth  estimate  for  151 
of  the  309  distinct  MDAPs  that  filed  at  least  one  SAR  during  FY  1969-  FY  2007,  a  bit  less 
than  half  of  the  total.  The  programs  for  which  we  have  a  PAUC  growth  estimate  do  not 
include  any  of  the  approximately  75  MDAPs  that  were  terminated  with  little  or  no  production. 
It  would  be  interesting  to  have  a  reasonable  PAUC  growth  estimate  relative  to  the  Milestone 
(MS)  ll/B* * 3  baseline  for  these  programs,  but  developing  such  estimates  would  require  far 
more  resources  than  were  available  for  this  study.  The  study,  then,  does  not  provide  a 
comprehensive  picture  of  cost  growth;  doing  so  was  not  its  intent.  The  question  asked  is 
whether  changes  in  acquisition  policy  and  process  over  time  have  visibly  had  an  influence 
on  PAUC  growth.  We  ask  that  question  for  MDAPs  that  passed  MS  ll/B  as  Acquisition 
Category  (ACAT)  I  programs  and  progressed  into  full  rate  production.  The  question  is  not 
explored  for  programs  that  were  cancelled  or  truncated. 

Appendix  A  of  McNicol  and  Wu  (2014)  describes  the  sources  of  the  PAUC  growth 
estimates  used  and  puts  the  MDAPs  for  which  a  PAUC  growth  estimate  was  available  in  the 
context  of  the  entire  population  of  MDAPs.  The  paper  also  provides  the  data  we  used  on  a 
compact  disc  (CD).  Unless  stated  otherwise,  PAUC  growth  here  means  PAUC  growth 
normalized  to  the  MS  ll/B  baseline  quantity. 


Effects  of  Management  Initiatives  on  the  Costs  and  Schedules  of  Defense  Acquisition  Programs,  Vol. 
I:  Main  Report.  More  recently,  Obaid  Younossi  et  al.  (2007),  in  Is  Weapon  System  Cost  Growth 
Increasing?  A  Quantitative  Assessment  of  Completed  and  Ongoing  Programs,  concluded  from  their 

careful  study  of  trends  in  development  cost  growth  that  “despite  the  many  acquisition  reform  and 
other  DoD  management  initiatives  over  the  years,  the  development  cost  growth  of  military  systems 
has  not  been  reduced.”  (Summary  page  xx).  References  to  the  previous  literature  are  provided  in 
both  McNicol  and  Younossi. 

3  DoDI  5000.2  issued  Oct.  23,  2000,  formally  established  Milestones  A,  B,  and  C  (in  place  of 
Milestones  I,  II,  and  III)  as  the  main  decision  points  for  an  MDAP.  Milestones  A,  B,  and  C  began  to  be 
used  somewhat  earlier  for  new  programs,  however. 
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The  second  major  challenge  was  one  of  research  design.  In  broad  outline,  the  paper 
identifies  natural  experiments  that  may  shed  some  light  on  the  effects  of  acquisition  policy 
on  PAUC  growth  and  then  interprets  the  outcomes  of  those  experiments.  In  part,  this  is 
straightforward.  We  know  when  the  main  changes  in  acquisition  policy  and  process 
occurred  and  what  they  were.  The  overall  DoD  acquisition  funding  climate  in  various 
periods — the  second  main  element  of  the  natural  experiment — also  can  be  readily 
established.  The  problem  is  that,  in  addition  to  the  easily  identified  elements  of  the  natural 
experimental  design,  there  are  a  considerable  number  of  other  factors  that  had  some 
influence  on  PAUC  growth.  We  first  limit  attention  to  acquisition  regime  and  funding  climate 
and  then,  as  particular  results  are  stated,  ask  whether  they  are  compromised  by  the 
omission  of  other  factors. 

Building  Blocks 

Discussions  of  acquisition  reform  over  the  past  25  years  have  usually  put  DoD 
program  manager  (PM)  and  personnel  in  the  program  office  in  the  foreground.  These  people 
oversee  the  contractors  and  do  a  myriad  of  things  that  must  be  done  by  the  government  for 
a  major  acquisition  program  to  move  forward — contracting,  financial  management,  and  test 
planning,  among  many  others.  In  the  background  are  the  contractors  who  typically  do  the 
development  and  manufacturing.  A  good  program  will  not  occur  if  the  government  personnel 
and  contractors  do  not  do  their  jobs  well.  It  is  equally  true  that  if  these  individuals  and 
organizations  do  their  jobs  well,  a  good  outcome  for  the  program  is  more  likely. 

What  this  focus  on  the  DoD  PM,  the  program  office  personnel,  and  the  contractors’ 
PMs  and  workers  leaves  out  are  factors  they  must  accept  as  “givens.”  These  givens  are 
subject  to  changes — sometimes  large  and  fairly  sudden — that  presumably  have  substantial 
consequences  for  program  outcomes.  One  of  the  givens  is  the  topline  DoD  funding 
constraint,  which  does  not  determine,  but  generally  has  a  marked  influence  on,  the  funding 
for  individual  MDAPs.  A  second  is  DoD  acquisition  policy  and  process.  We  begin  with  the 
latter. 

Acquisition  Regimes 

This  paper  distinguishes  five  successive  DoD  acquisition  regimes:4 

1.  The  Defense  Systems  Acquisition  Review  Council  (DSARC),  1970-1982 

2.  The  Post-Carlucci  Initiatives  DSARC,  1 983-1 9895 

3.  The  Defense  Acquisition  Board  (DAB),  1990-1993 

4.  Acquisition  Reform  (AR),  1994-2000 


4  The  main  reference  we  have  used  is  J.  Ronald  Fox  (2011),  Defens e  Acquisition  Reform,  1960  to 
2009:  An  Elusive  Goal.  Fox  identifies  the  main  features  of  each  of  these  periods  as  well  as  the  most 
important  changes  that  took  place  within  them. 

5  After  then  Deputy  Secretary  of  Defense  Frank  Carlucci.  There  is  some  uncertainty  about  when  the 
Post-Carlucci  Reforms  DSARC  should  end  and  the  DAB  regime  should  begin.  The  relevant  statutes 
were  passed  in  1986,  and  the  DAB  began  functioning  under  that  name  in  late  FY  1987  or  early  FY 
1988;  however,  the  DoD  did  not  implement  the  full  set  of  reforms  required  by  statute  until  1990.  We 
have  for  that  reason  set  the  line  at  1990. 
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5. 


The  DAB-Post  Acquisition  Reform,  2001-2007  (because  our  PAUC  growth 
data  ends  in  2007) 

The  transition  from  the  first  phase  of  the  DSARC  (1970-1982)  to  the  second  was 
principally  a  matter  of  policy  direction  and  renewal.  The  34  Carlucci  Initiatives  (regime 
number  2)  were  intended  to  increase  the  efficiency  and  effectiveness  of  the  OSD  acquisition 
oversight  process  and  the  Planning,  Programming,  and  Budgeting  System  (PPBS).  While 
the  DAB  (number  3)  itself  bears  a  strong  family  resemblance  to  the  DSARC,  the  statute 
creating  it  directed  management  changes  intended  to  strengthen  what  is  now  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]).  This  statute 
also  created  the  position  of  Vice  Chairman  of  the  Joint  Chiefs  of  Staff  (VCJCS)  and  directed 
a  new  requirements  process  centered  on  the  VCJCS.  The  results  sought  by  AR  (number  4) 
were  improvements  in  MDAP  outcomes,  but  changes  made  during  that  period  (not  all  part  of 
AR)  somewhat  relaxed  OSD  oversight  of  MDAPs.  There  were,  in  particular,  substantial  cuts 
in  acquisition  staffs  at  both  the  OSD  level  and  Service  Headquarters  level,  and  senior 
decision-makers  took  a  more  permissive  attitude  towards  cost  growth.  The  Post-AR  regime 
(number  5)  was  marked  by  the  arrival  of  a  new  administration  in  January  2001,  which 
brought  policy  changes  but  no  major  changes  to  the  acquisition  process  or  statutes. 

Table  1  displays  the  average  PAUC  growth  for  MDAPs  that  passed  MS  ll/B  or  filed  a 
first  SAR  in  each  of  these  successive  regimes.  There  are  a  number  of  interesting  aspects  to 
these  data,  for  example,  the  high  PAUC  growth  during  the  AR  period  and  the  lower  PAUC 
growth  for  FY  2001-FY  2007.  Granting  that,  the  single  most  notable  feature  of  these  data  is 
the  absence  of  any  trend  in  PAUC  growth.  If  changes  in  acquisition  policy  and  process  have 
had  a  sustained  influence  on  PAUC  growth,  it  does  not  show  up  in  this  table. 

Table  1.  Average  PAUC  Growth  in  Successive  Acquisition  Regimes 


Average  PAUC  No.  of 

Acquisition  Regime  Time  Period  Growth  Observations 


DSARC 

1970-1982 

32% 

48 

Post-Carlucci  Initiatives  DSARC 

1983-1989 

19% 

40 

DAB 

1990-1993 

36% 

11 

Acquisition  Reform  (AR) 

1994-2000 

66% 

27 

DAB  post  AR 

2001-2007 

19% 

25 

In  constructing  Table  1,  we  assigned  the  PAUC  growth  of  each  program  to  the 
acquisition  regime  in  place  when  the  program  passed  MS  ll/B  or  filed  its  first  SAR.  At  first 
glance  this  may  seem  dubious,  since  a  program  can  easily  take  10  or  15  years  from  the  start 
of  Engineering  and  Manufacturing  Development  (EMD)  through  delivery  of  the  final 
production  lot  and  thus  spend  parts  of  its  acquisition  cycle  under  successive  acquisition 
regimes.  Note,  however,  that  our  estimates  are  of  PAUC  growth  measured  from  the  baseline 
established  at  MS  ll/B,  which  does  not  change  over  the  course  of  a  program’s  acquisition 
cycle.  It  remains  possible  that  the  actual  acquisition  costs  of  a  program  are  significantly 
influenced  by  policy  or  process  changes  made  after  its  MS  ll/B.  Evidence  presented  in 
Appendix  B  of  McNicol  and  Wu  (2014)  suggests  that  if  such  influences  exist,  they  are  much 
smaller  than  the  effect  of  the  cost  estimate  in  the  MS  ll/B  baseline. 

Broadly,  there  are  two  ways  to  explain  the  absence  of  sustained  effects  of  acquisition 
policy  and  process  on  the  PAUC  growth  data.  First,  they  may  in  fact  not  have  a  strong  or 
consistent  effect  on  PAUC  growth.  Second,  acquisition  policy  and  process  may  have 
substantial  effects  that  are  masked  by  some  other  factor  or  factors. 
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Funding  Climate 

Thinking  along  the  lines  of  the  second  of  these  possibilities  led  to  consideration  of 
whether  changes  in  the  DoD  acquisition  funding  climate  might  be  associated  with  PAUC 
growth.  We  have  a  PAUC  growth  estimate  for  151  MDAPs  that  passed  MS  ll/B  or  submitted 
a  first  SAR  during  FY  1969-FY  2007.  This  period  includes  two  sub-periods  during  which 
acquisition  funding  was  Relatively  Constrained:  FY  1970-FY  1980  and  FY  1987-FY  2002. 
We  also  have  two  sub-periods  in  which  the  acquisition  funding  climate  was  Relatively 
Accommodating:  FY  1981-FY  1986  and  FY  2003-FY  2007.  (We  did  not  include  any 
program  not  at  least  five  years  beyond  MS  ll/B,  and  the  2012  SARs  were  the  last  available 
for  this  study.)6 

Table  2  displays  the  average  PAUC  growth  data  for  these  four  sub-periods.  The 
average  PAUC  growth  in  periods  of  Relatively  Constrained  acquisition  funding  is  far  larger 
than  it  is  in  periods  of  a  Relatively  Accommodating  funding  climate — by  a  factor  of  three  in 
the  first  comparison  and  by  a  factor  of  seven  in  the  second. 


Table  2.  Average  PAUC  Growth  During  Different  Acquisition  Funding  Climates 


DoD  Topline  Relatively  Constrained 

DoD  Funding  Relatively  Accommodating 

Period  (FY) 

PAUC  Growth 

Period  (FY) 

PAUC  Growth 

1970-1980 

35%  (42) 

1981-1986 

12%  (35) 

1987-2002 

53%  (55) 

2003-2007 

7%  (19) 

Note  Numbers  in  parentheses  are  the  number  of  observations  available. 


What  the  data  in  Table  2  portray  goes  well  beyond  “budget  instability”  as  usually 
understood.  Budget  instability  is  a  term  of  art  for  changes  in  MDAP  funding  through  the 
annual  resourcing  cycle  and  “taxes.”  Budget  instability  is  a  chronic  condition,  present  to 
some  degree  in  all  periods.  What  this  paper  observed  is  a  recurring  pattern — that  MDAPs 
that  passed  MS  ll/B  during  periods  when  the  acquisition  funding  was  Relatively  Constrained, 
on  average,  had  much  higher  PAUC  growth  than  those  that  passed  MS  ll/B  during  periods 
of  a  Relatively  Accommodating  funding  climate. 

Statistical  Results 

Table  3  expands  Table  2  by  replacing  the  funding  climate  sub-periods  with  the 
acquisition  policy  and  process  regimes.  This  table  provides  results  for  two  sets  of  natural 
experiments.  First,  the  PAUC  growth  columns  give  the  effect  of  changes  in  the  acquisition 
regime  for  a  given  funding  climate.  Second,  the  rows  show  the  effect  of  funding  climate  for  a 
given  acquisition  regime.  For  example,  the  first  11  years  of  the  DSARC  (FY  1970-FY  1980) 


6  We  use  as  the  breakpoints  that  define  these  periods  events  that  marked  major  changes  in 
expectations  about  the  course  of  defense  spending:  (1 )  the  invasion  of  Afghanistan  by  the  Soviet 
Union  in  December  1979  (FY  1980),  which  about  a  month  later  led  President  Carter  to  announce  a 
policy  of  sustained  increases  in  defense  spending  starting  with  the  FY  1981  funding;  (2)  the  adoption 
in  December  1 985  (end  of  the  first  quarter  of  FY  1 986)  of  the  Gramm-Rudman-Hollings  Act,  the 
funding  constraints  of  which  effectively  ended  the  Carter-Reagan  defense  buildup:  and  (3)  the 
9/1 1/2001  attacks.  It  is  important  to  recognize  that  key  decisions  made  within  the  DoD  on  content, 
costing,  and  funding  for  particular  MDAPs  in  a  given  year  are  made  at  least  a  year  in  advance  of  the 
submission  of  the  funding  for  that  year  to  the  Congress.  Consequently,  the  DoD  decisions  in  funding 
submissions  reflect  expectations  about  the  climate  that  submission  will  encounter. 
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were  in  a  tight  funding  climate,  while  the  next  two  (FY  1981-FY  1982)  were  in  a  period  in 
which  the  acquisition  funding  climate  was  Relatively  Accommodating. 

The  two  sections  that  follow  discuss,  in  turn,  whether  changes  in  acquisition  policy 
and  process  have  visible  effects  on  PAUC  growth,  and  the  association  between  funding 
climates  and  PAUC  growth. 


Table  3.  Average  PAUC  Growth  by  Acquisition  Regime  and  Funding  Climate 


Acquisition  Regime 

Relatively  Constrained 

Relatively  Accommodating 

Period  (FY) 

PAUC  Growth 

Period  (FY) 

PAUC  Growth 

DSARC 

1970-1980 

35%  (42) 

1981-1982 

11%  (6) 

Post-Carlucci  DSARC 

1987-1989 

34%  (11) 

1983-1986 

13%  (29) 

DAB 

1990-1993 

36%  (11) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

66%  (27) 

None 

N/A 

DAB  post  AR 

2001-2002 

57%  (6) 

2003-2007 

7%  (19) 

Note.  Numbers  in  parentheses  are  the  number  of  observations  available. 


Any  Trend  in  PAUC  Growth? 

There  is  no  doubt  that  direction  from  the  Milestone  Decision  Authority  (MDA) 
changes  particular  MDAPs,  and  some  of  those  changes  reduce  the  risks  of  major  PAUC 
growth  or  other  program  performance  shortfalls.  Viewed  from  this  perspective,  the  question 
asked  here  is  whether  the  decisions  made  (or  not  made)  in  different  acquisition  regimes  are 
large  enough  and  frequent  enough  to  be  visible  in  average  PAUC  growth. 

Looking  first  at  the  Relatively  Accommodating  climate  (column  on  the  right  in  Table 
3)  and  recognizing  that  it  is  likely  that  the  average  PAUC  growth  for  FY  2003-FY  2007 
eventually  will  be  a  few  percentage  points  higher,7  we  can  see  no  trend  towards  reduction  in 
average  PAUC  growth  in  periods  with  a  Relatively  Accommodating  funding  climate. 
Statistical  analysis  is  consistent  with  this  impression;  that  is,  we  found  no  evidence  of 
statistically  significant  differences  among  average  PAUC  growth  rates  for  the  Relatively 
Accommodating  funding  climate.8 


7  We  have  a  PAUC  growth  estimate  for  19  of  the  25  MDAPs  that  passed  MS  B  as  ACAT  I  programs 
during  2003-2007  and  which  have  not  been  cancelled  or  truncated.  Of  these  19  programs,  six  have 
been  completed,  six  are  in  full  rate  production  (FRP),  three  are  in  Low  Rate  Initial  Production  (LRIP), 
and  four  are  in  EMD.  Younossi  et  al.  (2007),  in  Is  Weapon  System  Cost  Growth  Increasing?,  found 
that  on  average,  60%  of  development  cost  growth  occurs  by  five  years  after  MS  B.  (31).  Estimates  of 
procurement  cost  growth  also  usually  are  increased  as  necessary  to  reflect  EMD  experience.  Since 
each  of  the  1 9  programs  is  at  least  five  years  beyond  MS  B,  even  a  doubling  of  the  7%  average 
PAUC  growth  would  be  unexpected. 

8  The  method  used  was  one-way  analysis  of  variance  (ANOVA).  ANOVA  is  a  test  of  whether  three  or 
more  samples  are  drawn  from  populations  with  the  same  mean.  The  null  hypothesis  is  that  all 
population  means  are  equal;  the  alternative  hypothesis  is  that  at  least  one  mean  is  different.  In  this 
case,  the  alternative  hypothesis  was  rejected  at  the  1%  level.  ANOVA  assumes  that  (1)  the 
populations  from  which  the  samples  were  drawn  are  normally  distributed,  (2)  the  samples  are 
independent,  and  (3)  the  variances  of  the  populations  are  equal.  We  are  grateful  to  Dr.  Sarah  Burns 
for  her  advice  on  the  statistical  analysis  and  for  doing  the  computations. 
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The  average  PAUC  growth  rates  for  the  two  most  recent  acquisition  regimes  during  a 
Relatively  Constrained  funding  (column  on  the  left  in  Table  3)  are  noticeably  larger  than 
those  for  the  three  earlier  periods.  Again,  however,  the  statistical  analysis  did  not  indicate 
that  any  of  the  averages  is  statistically  different  from  the  others  at  the  1%  confidence  level. 

Appendix  C  of  McNicol  and  Wu  (2014)  presents  a  table  similar  to  Table  3  for  each  of 
the  Military  Departments  and  for  joint  programs.  None  shows  an  improving  trend  in  PAUC 
growth  in  either  of  the  two  funding  climates,  and  with  a  small  number  of  exceptions,  these 
tables  show  the  same  features  we  see  in  Table  3. 

We  have  no  fully  comparable  PAUC  growth  data  for  the  periods  before  the  DSARC 
was  established.  Consequently,  the  statistical  analysis  leaves  open  the  possibility  that  the 
DSARC  and  its  successors  provided  a  useful  discipline  on  acquisition  programs.9  Moreover, 
the  statistical  analysis  does  not  erase  history.  Weapon  system  cost  growth  was  a  particular 
concern  during  the  1980s — the  sixth  Carlucci  Initiative  was  “Funding  to  Most  Likely  Costs” — 
and  it  is  reasonable  to  believe  that  the  Carlucci  Initiatives  did  in  fact  lead  to  more  vigorous 
enforcement  of  realistic  funding.  We  also  know  that  less  emphasis  was  placed  on  weapon 
system  cost  growth  during  the  AR  years,  and  oversight  of  acquisition  programs  was 
somewhat  more  relaxed.  It  could  be  that  these  differences  do  provide  part  of  the  explanation 
for  the  higher  observed  average  PAUC  growth  during  the  AR  years.  The  statistical  analysis 
prevents  us  from  asserting  with  confidence  that  they  do,  however,  because  those 
differences  are  within  the  bounds  of  what  can  be  expected  from  the  variability  of  the  data.10 

It  remains  possible  that  factors  that  have  not  been  considered  in  this  paper  mask 
significant  influences  of  acquisition  policy  and  process  on  PAUC  growth  that  a  more  refined 
analysis  would  reveal.* 11  In  considering  this  possibility,  it  must  be  kept  in  mind  that  the  issue 
is  growth  in  PAUC  from  the  MS  ll/B  baseline,  not  the  trend  over  time  in  costs  for  a 
commodity  group  or  the  acquisition  portfolio  as  a  whole.  Such  trends,  which  may  well  exist, 
do  not  necessarily  imply  more  cost  growth  for  individual  programs,  as  they  should  be 


9  The  most  nearly  comparable  data  seem  to  be  those  in  Table  A-7  (pp.  A-6  to  A-8)  of  Appendix  A  of 
Tyson  et  al.  (1992),  The  Effects  of  Management  Initiatives,  Vol.  I:  Main  Report.  These  data  are 
quantity  adjusted,  but  for  some  programs  cost  growth  may  not  be  measured  from  the  estimates  at  the 
start  of  EMD,  and  the  sample  may  include  programs  that  were  cancelled.  Omitting  one  program  with 
an  extremely  high  cost  growth  (Condor),  the  average  PAUC  growth  for  the  pre-1970  MDAPs  was 
48%,  which  is  noticeably  higher  than  the  average  PAUC  growths  during  1970-1978. 

10  For  example,  the  exceptionally  high  average  PAUC  growth  during  the  AR  years  (66%)  can  be 
attributed  in  part  to  changes  in  the  program  mix.  During  the  AR  years,  four  helicopter  programs 
passed  MS  ll/B,  one  more  than  average  for  a  period  of  this  length.  Moving  the  helicopter  program 
with  the  highest  PAUC  growth  (H-1  Upgrades)  from  1994-2000  to  1987-1989  reduces  the  average 
PAUC  growth  for  the  AR  years  from  66%  to  61%,  and  increases  the  average  for  1987-1989  from 
34%  to  48%. 

11  Results  of  McNicol  (2004)  point  to  one  possibility — changes  in  programs  that  occur  after  they  pass 
MS  ll/B.  This  work  considered  growth  in  quantity  normalized  unit  procurement  costs  after  excluding 
costs  of  unforced  changes  in  program  content  (i.e.,  changes  not  required  to  overcome  some  problem 
“baked  into”  the  program  in  the  MS  ll/B  baseline).  McNicol  found  evidence  that  some  changes  in  the 
acquisition  process  had  had  a  statistically  significant  effect  on  this  measure  of  cost  growth.  See,  in 
particular,  pp.  43-44  and  55-56. 
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reflected  in  the  MS  ll/B  baselines.  Further  comments  on  possible  confounding  variables  are 
made  below.  The  conclusion  offered  here  is  that  once  we  normalize  for  funding  climate,  we 
do  not  observe  any  improvement  in  PAUC  growth  from  the  changes  made  over  the  years  in 
acquisition  policy  and  process. 

Association  of  Funding  Climate  and  PAUC  Growth 

Returning  to  Table  3,  the  relevant  comparisons  are  between  the  Relatively 
Constrained  and  Relatively  Accommodating  funding  climates  for  a  given  acquisition  regime. 
Whereas  the  effects  on  PAUC  growth  of  the  different  acquisition  regimes  are  elusive,  those 
of  the  contrasting  funding  periods  stand  out  sharply. 

We  have  only  three  experiments  of  changes  in  funding  climate  for  a  given  acquisition 
regime,  since  two  of  the  five  acquisition  regimes  (DAB  and  AR)  fall  entirely  within  one 
funding  climate — Relatively  Constrained.  Each  of  these  three  natural  experiments  on  the 
effect  of  funding  climate  has  the  same  outcome — passing  MS  ll/B  during  a  Relatively 
Constrained  funding  climate  is  on  average  associated  with  much  higher  PAUC  growth 
compared  to  passing  during  a  Relatively  Accommodating  funding  climate  for  a  given 
acquisition  regime.  The  outcomes  of  the  first  two  experiments  are  virtually  identical — an 
average  PAUC  growth  of  35%  and  34%,  respectively,  in  the  two  periods  when  the  topline 
was  Relatively  Constrained  and  average  PAUC  growth  of  11%  and  13%,  respectively,  in  the 
two  periods  when  the  topline  was  Relatively  Accommodating.  The  effect  seems  most 
pronounced  in  the  third  experiment  (DAB  post-AR) — 57%  for  FY  2001-FY  2002  and  7%  for 
FY  2003-FY  2007.  (As  noted  earlier,  PAUC  growth  for  the  later  period  will  increase 
somewhat  as  the  programs  of  that  period  are  completed;  see  footnote  8.)  The  statistical 
analysis  found  each  of  these  differences  to  be  significant  at  more  than  the  1%  level.12 

There  is  a  distinct  pattern  to  the  changes  in  funding  climate  over  our  sample  period — 
bust,  boom,  bust,  boom.  If  some  other  factor  or  combination  of  factors  is  actually  at  work, 
rather  than  funding  climate,  it  would  have  to  have  this  same  pattern.  One  possibility  is  that 
the  methods  used  to  estimate  PAUC  growth  for  MDAPs  that  passed  MS  II  during  FY  1989- 
FY  2007  are  not  the  same  as  the  method  used  for  those  that  passed  MS  II  during  FY  1970- 
FY  1988.  Appendix  A  of  McNicol  and  Wu  (2014)  provides  a  comparison  that  suggests  that 
differences  in  estimating  methods  do  not  explain  the  low  average  PAUC  growth  recorded  for 
FY  2003-FY  2007  or  the  comparatively  high  cost  growth  observed  for  FY  1990-FY  2002. 
The  obvious  interpretation  of  Table  3,  and  the  one  we  believe  to  be  correct,  is  that  it  really  is 
changes  in  funding  climate  at  work. 

This  does  not  mean  that  a  Relatively  Constrained  funding  climate  causes  PAUC 
growth.  The  proximate  causes  of  PAUC  growth  are  decisions  embedded  in  programs 
approved  at  MS  ll/B  (unrealistic  cost  estimates  or  programmatic  assumptions,  for  example) 
and  decisions  made  during  program  execution  (such  as  failing  to  act  promptly  enough  on 
test  results)  that  eventually  lead  to  PAUC  growth.  The  correlation  observed  between  higher 
PAUC  growth  and  periods  of  tighter  funding  climate  does  suggest  that  programs  are  more 
likely  to  be  burdened  with  such  decisions  if  they  passed  MS  ll/B  during  a  Relatively 
Constrained  funding  climate. 


12  In  this  case,  we  used  the  usual  one-tail  test  for  the  difference  between  the  means  of  samples 
drawn  from  what  are  assumed  to  be  normal  populations. 
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Is  High  PAUC  Growth  Systemic  in  the  Relatively  Constrained  Funding 
Climate? 

Based  on  the  analysis  thus  far,  it  would  not  be  surprising  to  find  that  almost  all 
programs  that  pass  MS  ll/B  during  a  period  with  a  Relatively  Constrained  funding  climate 
are  burdened  with  the  sorts  of  very  optimistic  programmatic  and  costing  assumptions  that 
tend  to  result  in  high  PAUC  growth.  Alternatively,  the  bulk  of  the  cost  growth  might  be 
accounted  for  by  a  relatively  small  number  of  MDAPs.  Which  of  these  two  cases  is  the  more 
nearly  accurate  is  relevant  to  the  discussion  of  acquisition  reform.  In  the  first  case,  it  is 
reasonable  to  assume  that  PAUC  growth  is  a  systemic  problem.  It  is  often  said,  for  example, 
that  the  acquisition  culture  has  a  bias  in  favor  of  optimistic  programmatic  and  cost 
assumptions.  PAUC  growth  looks  much  less  like  a  systemic  problem  with  the  acquisition 
process,  however,  if  most  of  it  is  due  to  a  small  number  of  MDAPs. 

The  first  column  of  Table  4  shows  the  average  PAUC  growth  (in  periods  of  Relatively 
Constrained  funding  climate)  of  MDAPs  that  had  a  PAUC  growth  between  zero  and  50%. 
The  average  for  these  MDAPs  was  a  PAUC  growth  of  about  22%.  The  second  column 
shows  the  average  PAUC  growth  of  those  MDAPs  that  experienced  a  PAUC  growth  of  at 
least  50%.  These  range  from  a  low  of  71%  (FY  1970-FY  1980)  to  a  high  of  122%  (FY 
1994-FY  2000).  The  average  of  these  values  is  94%.  Finally,  the  last  column  in  Table  4 
shows  the  percentage  of  PAUC  growth  in  these  periods  accounted  for  by  MDAPs  with 
PAUC  growth  of  at  least  50%.  The  range  is  62%  to  89%  and,  averaged  across  all  five 
periods,  the  high  cost  growth  MDAPs  accounted  for  just  over  three-quarters  of  total  PAUC 
growth.  (The  figures  shown  in  Table  4  are  computed  from  simple  averages  rather  than 
weighted  by  program  size.13 )  In  short,  PAUC  growth  is  mainly  an  affliction  of  Relatively 
Constrained  funding  climates  and  it  is  primarily  due  to  a  minority  of  programs — on  the  order 
of  37% — that  experience  PAUC  growth  of  upwards  of  50%. 


Table  4.  Characteristics  of  PAUC  Growth  in  Relatively  Constrained  Funding 

Climate 


Acquisition  Regime 

Period 

(FY) 

Average  PAUC 
Growth  of 
MDAPs  with 
PAUC  Growth 
between 

0%  and  50% 

Average  PAUC 
Growth  of 
MDAPs  with 
PAUC  Growth 

>  50% 

%  of  PAUC 
Growth 

Accounted  for 
by  MDAPs  with 
PAUC  Growth 
>  50% 

DSARC 

1970-1980 

21%  (22) 

71%  (15) 

73% 

Post-Carlucci  DSARC 

1987-1989 

22%  (7) 

117%  (2) 

62% 

DAB 

1990-1993 

21%  (7) 

84%  (3) 

64% 

Acquisition  Reform 

1994-2000 

22%  (10) 

122%  (13) 

89% 

DAB  post  AR 

2001-2002 

29%  (3) 

85%  (3) 

75% 

Wore  Numbers  in  parentheses  are  the  number  of  observations. 


13  Weighting  by  program  size  would  be  required  in  any  consideration  of  the  effect  of  PAUC  growth  on 
funding,  because  cost  growth  on  a  large  program  has  a  greater  effect  on  funding  requirements  than 
cost  growth  of  the  same  magnitude  on  a  smaller  program.  This  paper,  however,  is  concerned  with 
examining  the  extent  to  which  PAUC  growth  is  associated  with  particular  combinations  of  acquisition 
regimes  and  funding  climates  and  in  such  a  context,  each  observation  counts  as  much  as  any  other. 
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Does  the  Resource  Allocation  Process  Play  a  Major  Role  in  PAUC  Growth? 

This  section  turns  to  a  discussion  of  MDAPs  that  experienced  very  high  cost  growth 
(interpreted  as  a  PAUC  growth  of  at  least  50%)  and  MDAPs  that  experienced  negative  cost 
growth.  Investigation  of  this  topic  was  initially  prompted  by  the  prospect  that  instances  of 
high  cost  growth  and  perhaps  also  of  negative  cost  growth  mask  effects  of  acquisition  policy 
and  process  on  PAUC  growth.  The  topic  proves  to  be  interesting  for  other  reasons  as  well. 
First,  it  provides  clear  and  unexpected  evidence  of  the  connection  between  PAUC  growth 
and  funding  climate  and,  by  implication,  the  DoD  resource  allocation  process.  Second,  cost 
growth  proves  not  to  be  a  problem  with  the  typical  system  but  with  the  minority  of  MDAPs 
that  experience  very  high  cost  growth. 

Negative  PAUC  Growth 

Twenty-nine  MDAPs  in  our  sample  show  negative  PAUC  growth  (not  including  four 
cases  of  zero  PAUC  growth).  Viewed  from  an  acquisition  perspective,  negative  PAUC 
growth  seems  anomalous;  in  fact,  it  is  not  uncommon  to  hear  confident  assertions  to  the 
effect  that  MDAPs  never  underrun  their  funding.  It  is  understandable,  however,  in  a  resource 
allocation  context  at  the  Service  level. 

Negative  PAUC  growth  is  recorded  if  the  actual  cost  of  a  program  proves  to  be  less 
than  the  cost  in  the  MS  ll/B  baseline.  Negative  PAUC  growth  can  occur  because  a  program 
was  particularly  well  managed  or  lucky.  It  also  can  occur  if  the  ambitions  of  a  program  are 
scaled  back  after  a  program  has  passed  MS  ll/B.  In  addition,  negative  PAUC  growth  can 
grow  out  of  resource  allocation  imperatives. 

Assuming  the  program  was  funded  to  its  MS  ll/B  baseline,  negative  PAUC  growth 
implies  that  over  time  funds  can  be  taken  from  the  program  in  question  and  reallocated  to 
other  applications,  including  other  acquisition  programs.  The  program,  then,  effectively  can 
be  used  as  a  “bank” — a  way  to  hold  reserves  in  relative  safety  until  they  are  needed.  A 
“withdrawal”  can  be  made  in  the  execution  year  with  the  approval  of  the  Congress,  but  for 
the  outyears  of  the  Future  Years  Defense  Plan  (FYDP),  the  Service  can  simply  initiate  the 
reallocation  in  its  Program/Funding  submission  to  the  OSD. 

A  bank  of  this  sort  is  more  likely  to  be  needed  in  a  Relatively  Accommodating 
funding  climate,  as  it  can  then  serve  as  a  way  to  delay  final  decisions  on  allocation  of  the 
higher  level  of  funding  that  has  become  available.  We  would  therefore  expect  to  find 
relatively  more  instances  of  negative  PAUC  growth  in  the  Relatively  Accommodating  funding 
periods,  and  this  is  what  we  observe.  As  the  data  in  Table  5  indicate,  about  30%  of  our 
observations  in  Relatively  Accommodating  funding  climates  are  of  negative  PAUCs, 
compared  to  about  12%  across  the  periods  of  Relatively  Constrained  climate. 


Table  5.  Number  of  PAUC  Growth  Observations  Less  Than  Zero  by  Acquisition 

Regime  and  Funding  Climate 


Acquisition  Regime 

Topline  Relatively 
Constrained 

Topline  Relatively 
Accommodating 

Time  Period 
(FY) 

PAUC  Growth 
<0 

Time  Period 
(FY) 

PAUC  Growth 
<0 

DSARC 

1970-1980 

5  of  42 

1981-1982 

2  of  6 

Post-Carlucci  DSARC 

1987-1989 

2  of  11 

1983-1986 

10  of  29 

DAB 

1990-1993 

1  of  11 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

4  of  27 

None 

N/A 

DAB  post  AR 

2001-2002 

Oof  6 

2003-2007 

5  of  19 
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Negative  PAUC  growth  is  not  regarded  as  a  problem,  probably  correctly.  It  is, 
however,  a  clear  and  unexpected  case  in  which  PAUC  growth  reflects  accommodation  to 
the  funding  climate. 

PAUC  Growth  >  50% 

Adoption  of  unrealistically  low  cost  estimates  at  MS  ll/B  creates  the  illusion  that  the 
funds  available  over  the  FYDP  and  beyond  will  support  more  MDAPs  than  they  in  fact  will. 
That  is,  unrealistically  optimistic  costing  will  for  a  time  permit  more  new  starts.14  In  addition, 
the  conventional  wisdom  holds  that  a  lower  MS  ll/B  cost  makes  it  easier  to  gain  the 
concurrence  of  the  OSD,  the  Office  of  Management  and  Budget,  and  the  Congress  for  a 
new  program.  Consequently,  we  would  expect  to  find  relatively  more  programs  with  PAUC 
growth  of  at  least  50%  in  Relatively  Constrained  funding  climates,  which  is  in  fact  what  we 
do  find. 


Table  6  reports  the  number  of  programs  with  an  average  PAUC  growth  of  at  least 
50%.  Of  the  54  programs  that  passed  MS  ll/B  in  a  Relatively  Accommodating  funding 
climate,  only  four  showed  PAUC  growth  of  at  least  50%.  In  contrast,  36  of  the  97  programs 
that  passed  MS  ll/B  in  a  Relatively  Constrained  funding  climate  showed  cost  growth  of  at 
least  50%.  This  is  to  say  that  the  frequency  of  MDAPs  with  a  PAUC  growth  of  at  least  50% 
is  much  lower  in  periods  when  the  topline  is  Relatively  Accommodating  than  in  a  Relatively 
Constrained  funding  climate — 7%  versus  37%. 


Table  6.  Number  of  PAUC  Growth  Observations  >  50%  by  Acquisition  Regime 

and  Funding  Climate 


Acquisition  Regime 

Topline  Relatively 
Constrained 

Topline  Relatively 
Accommodating 

Time  Period 
(FY) 

PAUC  Growth 
>  50% 

Time  Period 
(FY) 

PAUC  Growth 
>  50% 

DSARC 

1970-1980 

15  of  42 

1981-1982 

0  of  6 

Post-Carlucci  DSARC 

1987-1989 

2  of  11 

1983-1986 

3  of  29 

DAB 

1990-1993 

3  of  11 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

13  of  27 

None 

N/A 

DAB  post  AR 

2001-2002 

3  of  6 

2003-2007 

1  of  19 

Reexamination  of  Trends  in  PAUC  Growth 

The  circumstances  in  which  we  are  more  likely  to  see  very  high  PAUC  growth  and 
instances  of  negative  PAUC  growth  suggest  that  they  reflect  accommodations  to  different 
funding  climates.  In  other  words,  instances  of  high  PAUC  and  negative  PAUC  may  not 
reflect  the  normal  operation  of  the  acquisition  process.  On  this  basis,  Table  7  presents 
average  PAUC  growths  computed  excluding  observations  of  greater  than  or  equal  to  50% 
and  negative  values. 

The  statistical  analysis  of  the  data  in  Table  7  produces  the  same  conclusions  as  that 
of  Table  3  in  one  important  respect:  there  is  no  indication  of  statistically  significant 
differences  across  acquisition  regimes  within  a  funding  climate.  The  difference  between  the 


14  It  is  not  clear  that  doing  this  ever  makes  financial  sense  because  the  “loans”  created  by 
unrealistically  low  cost  estimates  eventually  must  be  made  good  one  way  or  another  at  an  implicit  but 
steep  interest  rate. 
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averages  in  the  two  funding  climates  for  the  DAB  post  AR  is,  however,  statistically  significant 
at  the  1%  level,  but  the  differences  for  the  other  two  cases  (DSARC  and  Post-Carlucci 
DSARC)  are  not  statistically  different  at  conventional  levels  of  significance. 


Table  7.  Average  PAUC  Growth  Excluding  Observations  >  50%  and  Negative 
Observations  by  Acquisition  Regime  and  Funding  Climate 


Acquisition  Regime 

Topline  Relatively 
Constrained 

Topline  Relatively 
Accommodating 

Time 

Period  (FY) 

%  of  PAUC 
Growth 

Time 

Period 

(FY) 

%  of  PAUC 
Growth 

DSARC 

1970-1980 

21%  (22) 

1981-1982 

22%  (4) 

Post-Carlucci  DSARC 

1987-1989 

22%  (7) 

1983-1986 

13%  (16) 

DAB 

1990-1993 

21%  (7) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

22%  (10) 

None 

N/A 

DAB  post  AR 

2001-2002 

29%  (3) 

2003-2007 

10%  (131 

Note  Numbers  in  parentheses  are  the  number  of  observations. 


Implications  for  Discussions  of  Acquisition  Reform 

This  paper  points  to  three  implications  for  a  discussion  of  acquisition  reform.  First, 
the  relevant  context  for  understanding  PAUC  growth  is  the  interface  between  the  acquisition 
process  and  the  resource  allocation  process.  The  crucial  evidence  behind  this  point  is  the 
strong  association  between  funding  climate  and  PAUC  growth.  Resource  managers  must 
think  in  terms  of  a  portfolio  of  programs  across  mission  areas  and  commodity  types,  and 
extending  from  efforts  in  the  technology  base  through  programs  nearing  the  end  of 
production.  When  a  program  is  completed,  it  opens  a  resource  “hole”  that  programs 
emerging  from  EMD  can  occupy.  In  turn,  programs  earlier  in  the  acquisition  cycle  can  move 
forward  as  well.  When  funding  for  acquisition  turns  down,  these  holes  get  smaller,  or  close 
entirely,  or  require  cuts  in  funding  for  ongoing  programs.  The  alternatives  available  in  this 
circumstance  are  all  undesirable — cancellations  of  programs,  delays  in  new  starts, 
stretches,  and  unrealistic  costing.  The  evidence  summarized  here  suggests  that  it  is  in  this 
context  that  high  PAUC  growth  arises. 

Second,  it  seems  unlikely  that  further  changes  in  the  acquisition  process  would  have 
a  major  effect  on  PAUC  growth.  The  research  found  no  evidence  that  acquisition  policy  and 
process  changes  through  the  years  have  produced  sustained  and  significantly  lower  or 
higher  PAUC  growth.  This  does  not  mean  that  the  DAB  process  does  not  provide  a  useful 
discipline  on  acquisition  programs;  moreover,  further  changes  in  acquisition  policy  or 
process  might  be  warranted  for  reasons  of  good  government.  The  evidence  does,  at  a 
minimum,  suggest  that  the  effects  of  changes  in  the  acquisition  process  since  its 
inauguration  in  the  early  1970s  have  not  had  a  dominant  effect  on  PAUC  growth. 

Third,  it  is  difficult  to  see  that  the  cultures  of  the  DoD  acquisition  organizations  are  a 
crucial  obstacle  to  improved  performance  on  cost  growth.  The  key  point  to  note  is  that  high 
PAUC  growth  is  not  persistent,  but  rather  episodic,  and  correlated  with  environmental 
factors  outside  of  the  control  of  the  acquisition  process.  There  is  remarkably  little  PAUC 
growth  in  periods  when  the  funding  is  Relatively  Accommodating.  It  seems  fair  to  ask  if  it 
makes  sense  to  assert  that  an  entrenched  culture  sometimes  results  in  high  cost  growth  and 
other  times  in  low  cost  growth.  Just  how  is  it  that  the  A  team  takes  the  field  so  quickly  and 
quietly  when  the  budgetary  sun  comes  out?  And  why  even  in  bad  budgetary  weather  do 
more  than  half  of  MDAPs  exhibit  comparatively  modest  PAUC  growth? 
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